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(57) Abstract: The simulation of film grain in a video 
image occurs by first creating a block (i.e., a matrix 
array) of transformed coefficients for a set of cut fre- 
quencies fnL, fvL» fHH and fyn associated with a desired 
grain pattern. (The cut frequencies fuL* fvL> ^hh and 
fvH represent cut-off frequencies, in two dimensions, 
of a filter that characterizes the desired film grain pat- 
tern). The block of transformed coefficients undergoes 
an inverse transform to yield a bit-accurate film grain 
sample and the bit accurate sample undergoes scaling 
to enable blending with a video signal to simulate film 
grain in the signal. 
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LOW-COMPLEXITY FILM GRAIN SIMULATION TECHNIQUE 
CROSS-REFERENCE TO RELATED APPLICATIONS 

5 This application claims priority under 35 U.S.C. 1 19(e) to U.S. Provisional Patent 

Application Serial No 60/630640, filed November 23, 2004, the teachings of which are 
incorporated herein. 

TECHNICAL FIELD 

10 

This invention relates to a technique for simulating film grain in an image. 
BACKGROUND OF THE INVENTION 

15 Motion picture films comprise silver-halide crystals dispersed in an emulsion, which is 

coated in thin layers on a film base. The exposure and development of these crystals form the 
photographic image consisting of discrete tiny particles of silver. In color negatives, the silver 
undergoes chemical removal after development and tiny blobs of dye occw on the sites where 
the silver crystals form. These small specks of dye are commonly called 'grain' in color film. 

20 Grain appears randomly distributed on the resulting image because of the random formation 
of silver crystals on the original emulsion. Within a uniformly exposed area, some crystals 
develop after exposure while others do not. 

Grain varies in size and shape. The faster the film, the larger the clumps of silver 
formed and blobs of dye generated, and the more they tend to group together in random 

25 patterns. The grain pattern is typically known as 'granularity*. The naked eye cannot 

distinguish individual grains, which vary from 0.0002 mm to about 0.002 mm. Instead, the 
eye resolves groups of grains, referred to as blobs. A viewer identifies these groups of blobs 
as film grain. As the image resolution becomes larger, the perception of the film grain 
becomes higher. Film grain becomes clearly noticeable on cinema and high-definition 

30 images, whereas film grain progressively loses importance in SDTV and becomes 
imperceptible in smaller fomiats. 

Motion picture film typically contains image-dependent noise resulting either from the 
physical process of exposure and development of the photographic film or from the 
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subsequent editing of the images. The photographic film possesses a characteristic quasi- 
random pattern, or texture, resulting from physical granularity of the photographic emulsion. 
Alternatively, a similar pattern can be simulated over computed-generated images in order to 
blend them with photographic film. In both cases, this image-dependent noise is referred to as 
grain. Quite often, moderate grain texture presents a desirable feature in motion pictures. In 
some instances, the film grain provides visual cues that facilitate the correct perception of 
two-dimensional pictures. Film grain is often varied within a single film to provide various 
clues as to time reference, point of view, etc. Many other technical and artistic uses exist for 
controlling grain texture in the motion picture industry. Therefore, preserving the grainy 
appearance of images throughout image processing and delivery chain has become a 
requirement in the motion picture industry. 

Several conunercially available products have the capability of simulating film grain, 
often for blending a computer-generated object into a natural scene. Cineon® fi-om Eastman 
Kodak Co, Rochester New York, on^ of the first digital film applications to implement grain 
simulation, produces very realistic results for many grain types. However, the Cineon® 
application does not yield good performance for many high speed films becaxise of the 
noticeable diagonal stripes the application produces for high grain size settings. Further, the 
Cineon® application fails to simulate grain with adequate fidelity when images are subject to 
previous processing, for example, such as when the images are copied or digitally processed. 

Another commercial product that simulates film grain is Grain Surgery™ from Visual 
Infinity Inc., which is used as a plug-in of Adobe ® After Effects ®. The Grain Surgery™ 
product appears to generate synthetic grain by filtering a set of random numbers. This 
approach suffers from disadvantage of a higji computational complexity. 

None of these past schemes solves the problem of restoring film grain in compressed 
video. Film grain constitutes a high frequency quasi-random phenomenon that typically 
cannot imdergo compression using conventional spatial and temporal methods that take 
advantage of redundancies in the video sequences. Attempts to process film-originated 
images using MPEG-2 or ITU-T/ISO H.264 compression techniques usually result in either an 
unacceptably low degree of compression or complete loss of the grain texture. 

Thus, there exists a need for a technique simulating film grain, especially a technique 
that affords relatively low complexity. 
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BRIEF SUMMARY OF THE DSfVENTION 

Briefly, there is provided a method in accordance with the present principles for 
simulating film grain. The method commences by creating a block (i.e., a matrix array) of 
5 transformed coefficients responsive to a band pass region associated with a desired grain 

pattern. (The band pass region is defined by a set of cut firequencies fnu fvL, fm, and fvH that 
represent cut-off frequencies (in two dimensions) of a filter that characterizes the desired film 
grain pattern). The block of transformed coefficients undergoes an inverse transform to yield 
a bit-accurate film grain sample. If desired, the bit accurate sample can undergo scaling to 

10 enable blending with a video signal to simulate film grain in the signal. 

In practice, generation of the block of transformed coefficients occurs as follows. For 
each coefficient at coordinates {x,y) in the block, where x zsidy are integers designating the 
row and column, respectively, a random value (r) is obtained, if each of the x and y 
coordinates lies within the band pass of the cut frequencies frit and fHH and fvL and fvH, 

15 respectively. The value of the coefficient at the coordinates (x,y) becomes the random value 
r. The process is repeated for every entry in the block. 

The film grain simulation technique of the present principles affords reduced 
complexity as compared to prior- art transform-based approaches for film grain simulation by 
avoiding the computation of a direct transform. The technique also reduces the memory 

20 requirements of database*based approaches by storing a small set of transformed coefficients 
instead of film grain patterns. The present method has applicability to HD DVD systems, BD 
ROM systems as well as to satellite broadcasting, among other uses. 

DETAILED DESCRIPTION OF THE DRAWINGS 

25 

FIGURE 1 depicts a block schematic diagram of the combination of a transmitter and 
receiver in a film Grain Processing chain useful for practicing the technique of the present 
principles; 

FIGURE 2 depicts, in flow chart form, the steps of a first method for creating a set of 
30 transformed coefficients for simulating film grain in accordance with the present principles; 

FIGURE 3 depicts, in flow chart form, the steps of a second method for creating a set 
of transfomied coefficients for simulating film grain in accordance with the present principles; 
and 
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FIGURE 4 depicts, in flow chart fonn, the steps of a method for creating a database of 
film grain patterns in accordance with the present principles. 

DETAILED DESCRIPTION 

To best understand the technique of the present principles for simulating film grain 
using a set of cut fi-equencies that characterize the film grain pattern, a brief overview of film 
grain simulation will prove belpfiil. FIGURE 1 depicts a block schematic diagram of a 
transmitter 10, which receives an input video signal and, in turn, generates a compressed 
video stream at its output. Di addition, the transmitter 10 also generates infonnation indicative 
of the fihn grain (if any) present in the sample. In practice, die transmitter 10 could comprises 
part of a head-end array of a cable television system, or other such sj^tem that distributes 
compressed video to one or more downstream receivers 11, only one of which is shown in 
FIG. 1 . The transmitter 1 0 could also take the form of encoder that presents media like 
DVDs. The receiver 1 1 decodes the coded video stream and simulates film grain in 
accordance with the film grain information and decoded video, both received fi^om the 
transmitter 10 or directly fi-om the media itself in the case of a DVD or the like, to yield an 
output video stream that has simulated film grain. The receiver 1 1 can take the form of a set- 
top box or other such mechanism that serves to decode compressed video and simulate film 
grain in that video. 

The overall management of film grain requires the transmitter 10 (i.e., the encoder) 
provide information with respect to the film ^n in the incoming video. In other words, the 
transmitter 10 "models" the film grain. Further the receiver 1 1 (i.e., decoder) simulates die 
film grain according to the film grain information received firom the transmitter 10. The 
transmitter 10 enhances the quality of the compressed video by enabling the receiver 1 1 to 
simulate film grain in the video signal when difficulty exists in retaining the film grain during 
the video coding process. 

In the illustrated embodiment of FIG. 1, the transmitter 10 includes a video encoder 12 
which encodes the video stream using any of the well known video compression techniques 
such as the ITU-T Rec. H.264 | ISO/IEC 14496-10 video compression standard. Optionally, a 
film grain remover 14, in the form of a filter or the like depicted in dashed lines in FIG. 1, 
could exist upstream of the encoder 12 to remove any film grain in the incoming video stream 
prior to encoding. To the extent that the incoming video contains no film grain, no need 
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would exist for the film grain remover 14. 

A film grain modeler 16 accepts the input video stream, as well as the output signal of 
the film grain remover 14 (when present). Using such input information, the film grain 
modeler 16 establishes the film grain in the incoming video signal. In its simplest form, the 
5 film grain modeler 1 6 could comprise a look up table containing film grain models for 

different film stocks. Information in the incoming video signal would specify the particular 
film stock used to originally record the image prior to conversion into a video signal, thus 
allowing the film grain modeler 16 to select the appropriate film grain model for such film 
stock. Alternatively, the film grain modeler 16 could comprise a processor or dedicated logic 

10 circuit that would execute one or more algorithms to sample the incoming video and 
detemiine the film grain pattern that is present. 

The receiver 1 1 typically includes a video decoder 18 that serves to decode the 
compressed video stream received firom the transmitter 10. The structure of the decoder 18 
will depend on the type of compression performed by the encoder 12 within the transmitter 

15 10. Thus, for example, the use within the transmitter 10 of an encoder 12 that employs the 
ITU-T Rec. H,264 | ISO/EEC 14496-10 video compression standard to compress outgoing 
video will dictate the need for an H.264-compliant decoder 1 8. Within the receiver 1 1, a film 
grain simulator 20 receives the film grain information fi-om the film grain model 16. The film 
grain simulator 20 can take the form of a programmed processor, or dedicated logic circuit 

20 having the capability of simulating film grain for combination via a combiner 22 with the 
decoded video stream. 

Film grain simulation aims to synthesize film grain samples that simulate the look of 
the original film content. As described, film grain modeling occiuis at the transmitter 10 of 
FIG. 1, whereas film grain simulation occurs at the receiver 1 1. In particular, film grain 

25 simulation occurs in the receiver 1 1 along with the decoding the incoming video stream fi-om 
the transmitter 10 but upstream of the output of the decoded video stream. Note that the 
decoding process that occurs in the receiver 1 1 makes no use of images with added film grain. 
Rather, film grain simulation constitutes a post-processing method for synthesizing simulated 
film grain in the decoded images for display. For that reason, the ITU-T Rec. H.264 | 

30 ISO/DEC 14496-10 video compression standard contains no specifications regarding the film 
grain simulation process. However, film grain simulation requires information conceming the 
grain pattern in the incoming video signal, which information typically imdergoes 
transmission in a Supplemental Enhancement Information (SEI) message when using the ITU- 
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T Rec. H.264 | ISO/IEC 14496-10 video compression standard as specified by the 
Amendment 1 (Fidelity Range Extensions) of that compression standard. 

In accordance with the present principles, there is provided a low-complexity method 
for simulating film grain, which presupposes that the transformation used to simulate film 
5 grain is linear. The method of the present principles, which is practiced by the film grain 

simulator 20 of FIG; 1, generates a block of film grain in a process that includes the following 
three steps: (1 ) create a block of transformed coefficients in accordance with the cut- 
fi-equencies for a desired film grain pattern, (b) compute an inverse transform and (c) scale the 
resultant values: A more complete description of each process follows hereinafter. 

10 

Creating a block of transformed coefficients 

The process of creating of a block of transformed coefficients takes as its input a set of 
cut firequencies characterizing a desired film grain pattem and yields a set of normalized 

15 random numbers. In a preferred embodiment, the normalized random numbers will follow a 
normalized Gaussian random distribution (independent and identically distributed Gaussian 
random variable samples with zero mean and unity variance) while the cut fi^equencies are 
transmitted in a film grain characteristics SEI message, as defined by the ITU-T Rec. H.264 | 
ISO/IEC 14496-10 standard [1]. 

20 In the illustrated embodiment, the creation of a block image B (e.g., a block of 

transformed coefficients of size N x N) in the firequency domain occurs in the manner 
illustrated in flow chart form in FIG. 2. The process begins at Start step 100 during which 
initialization occurs. Following step 100, step 102 occurs, conmiencing a loop which scans all 
coefficients of the N x N block by setting a pair of running variables (x,y) to all possible 

25 values in the integer range of [0,N-1] x [0,N-1]. The variables x and y each represent the x 
and y coordinates, respectively, of an entry in a block (i.e., a matrix array) of transformed 
coefficients produced by this process. During step 104, a check occurs whether the values of 
X and y lie within the band pass defined by the cut frequencies fnL and f™ and fvL and fvH- If 
so, step 106 occurs and a random number is generated and during step 108 the random 

30 number is assigned to the coefficient in the block at the x and y coordinates matching the 
current value of the running variables x and y, respectively. Step 104 illustrates a preferred 
embodiment of the checking of x and y values to decide whether or not they lie within the 
band pass defined by the cut frequencies, but other criteria are possible. 
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If the cTirrent values of x and y do not lie simultaneously within the band pass defined 
by the cut frequencies fHL and Thh and fvL and fvH of the desired film grain pattern, as 
transmitted in the SEI message, then the coefficient in the block associated with the current 
value of X and y is set to zero during step 110. Following either of steps 108 or 1 10, then step 
112 occurs, and the running variables x and y are incremented, provided that x and y < N and 
execution branches to step 104. Otherwise, the loop ends during step 1 12 and the method 
finishes at End stq> 114. 

The method for creating a block of transformed coefficients depicted in FIG. 2 can be 
described in a "for" computer program loop in the following manner: 

for(y = 0;y<N;y++) 

for( X = 0; X < N; X-H-) 

if((X<fHL&&y<fvL) l|x>fHH II y>fvH) 

B[x][y]=0 

else 

B[x][y] = r 

B[0][0] = 0 

Note that fHL and fvt constitute the horizontal and vertical low cut firequencies, respectively; 
fHH and fvH constitute the horizontal and vertical high cut frequencies, respectively of the 
desired film grain pattern, and r constitutes a normalized random number. In practice, a 
Gaussian random number r could be generated from two independent, imiformly distributed 
random values, using the Box-Muller transformation specified by: 

r = ^~21n(xi) • sin(2;cc2) 

where ln(.) is the natural logarithm, sin(.) is the sine function, n is Archimedes's constant, and 
xi and X2 are in the range of (0,1), with 0 not included. 

FIGURE 3 depicts an alternate preferred method for creating a block of transformed 
coefficients. The flow chart method of FIG. 3 includes almost all of the same steps as the 
method of FIG. 2 and therefore like reference numbers appear in FIG. 3 as in FIG. 2 to 
describe like steps. The method of FIG. 3 differs in the following respect. Instead of 
obtaining a random number during step 106 in FIG. 2 through mathematical computation, the 



wo 2006/057703 



PCT/US2005/034723 



-8- 

method of creating a block of transformed coefficients of FIG. 3 includes the step 105 of 
obtaining a random number from a look-up table 1 10 of previously computed Gaussian 
random numbers. Otherwise, the two mediods practice the same steps. 

Optionally, the block transformed coefficient generation methods of FIGS. 2 and 3 
cpuld force the coefficient at the position 0,0 (i.e., the DC coefficient) to become zero in order 
to ensure that after inverse transform, the signal will have zero mean. Large deviations fiom 
zero for the DC coefficient could cause large deviations in the average value of the resulting 
film grain block. As a consequence, when blending the film grain with the decoded image, 
blocks could show visible variations in their average value (i.e., the blocks could become 
either darker or brighter). 

In the case when fnh and fvt both equal to zero, the creation of the block image B in 
the frequency domain can occur as follows: 

for(y = 0; y < N; y++ ) 

for( X = 0; X < N; X-H-) 

if(x>fHH II y>fvH) 
B[x][y] = 0 

else 

B[x][y] = r 

B[0KO]=O 

Assuming also the ability to accomplish zero value initialization upon memory allocation of 
the block image, the previous method can be simplified to the following expression: 

for( y = 0; y < N; y-H- ) 

for( X = 0; X < N; X++) 

ifl[x<=fHH && y<=fvH) 
B[x][y] = r 

Note that the processes described in FIGS. 2 and 3 make use of (fm - fHL+l)*(fvH - fvt + 1) 
normalized random values, thereby avoiding the filtering step that typically follows the 
computation of the forward transform of a block of normalized random.numbers. 
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Compute The Inverse Transform 

Following generation of the block of transformed coefficients, computation of the 
inverse transform occurs, typically according to the general formula: 

where a constitutes a scaling factor that compensates for intensity variation resulting from the 
generation of the block of transformed coefficients. Typically, the scaling factor a can have a 
value of unity. 

When using a DCT transform, computation of the transformation matrix C occurs as 
follows: 



k = l^^ ifm = 0 
\j2/N otherwise 



in this case, a = 



Scaling of the Inverse Transform Resultant Values 



Following the creation of the block of transformed coefficients and computation of the 
inverse transform of the block, the film grain simulation process of the present principles 
typically includes the step of scaling the film grain pattern to the appropriate film grain 
intensity, although such scaling not necessarily occur. For video coding applications, 
information about the film grain intensity typically undergoes transmission in the SEI 
message, as specified in the ITU-T Rec. H.264 | ISO/IEC 14496-10 standard. 
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The scaling process typically occurs in accordance in the following manner: 

for(y = 0;y<N;y-M-) 

for(x = 0; x <N;x-H-) 

b[ X ][ y ] *= scaling^factor 

This approach presents a tradeoff between complexity and memory requirements when 
compared to prior art methods. On one hand, the scaling method of the present principles 
reduces the complexity of transform^based approaches by avoiding computation of the direct 
transform and filtering in the frequency domain. On the other hand, the scaling method of the 
present principles reduces the memory requirements of database-based approaches by only 
storing a small set of transformed coefficients instead of complete film grain patterns. 

The method of the present principles can serve to initialize a database of film grain 
patterns. In particular, the steps of creating a the block of transformed coefficients and 
computing the inverse transform of the block could serve to initialize a database of film grain 
patterns as illustrated in flow chart form in Figure 4. The method of FIG. 4 commences upon 
execution of Start step 200 whereupon initialization occurs. Thereafter, step 202 occurs with 
generation of the block of Gaussian random values, using either the method of FIG. 2, or the 
method of FIG- 3. When employing the block transform generation method of FIG. 3, each 
random number entry is obtained by accessing the Gaussian random number look-up table 
107, as discussed previously. Following step 202, the block of Gaussian random values 
xmdergoes an inverse transform, typically via an Inverse Discrete Cosine Transfomi (IDCT) 
during step 204, along with the appropriate scaling. During step 206, the film grain patterns 
undergo storage in a database 207. Thereafter, the process ends at step 208. The process of 
FIG. 4 allows creation of the database 207 upon system initialization or reset, and enables 
storage in a conventional memory (usually SRAM memory). 

The foregoing describes a technique for simulating film grain in an image that affords 
the advantage of low complexity. 



wo 2006/057703 



PCTAJS2005/034723 



-11- 

CLAIMS 



1 1 . A method for simulating film grain, comprising the steps of: 

2 creating a block of transformed coefficients responsive to a band pass region 

3 associated with a desired film grain pattern; and 

4 establishing an inverse transform for the created block of transformed coefficients to . 

5 yield a film grain sample. 

1 2. The method according to claim 1 wherein the block of block transformed 

2 coefficients is created by the steps of: 

3 (a) obtaining a value for each coefficient at coordinates (x, y) which is a random value 

4 for so loiig as x and y lie in the band pass defined by a set of cut frequencies fnu fvu fiiH and 

5 fvH, but a zero otherwise; 

6 (b) setting the value of the coefficient at coordinates (x, y) to the obtained value; and 

7 (c) repeating steps (a) and (b) until a value is set for every coefficient in the block. 

1 3, The method according to claim 2 further comprising the step of setting the 

2 block coefficient at (0,0) to a zero value. 

1 4, The method according to claim 2 wherein each of the random values obtained for so 

2 long as x and y lie in the band pass defined by a set of cut frequencies fHL» fvL» ftm and fvH 

3 follows a Gaussian random distribution. 
1 

1 5. The method according to claim 2 wherein each of the random values is 

2 obtained by accessing an entry fi-om a pre-computed look-up table of random values. 
1 

1 6. The method according to claim 1 further including the step of scaling the film 

2 grain sample. 
1 

1 7. A method of creating a block of transformed coefficients used to simulate a 

2 desired film grain pattern, comprising the steps of: 

3 (a) obtaining a value for each coefficient at coordinates (x, y) which is a random value 

4 for so long as x and y lie in a band pass region defined by the cut firequencies fnu fvu fiiH and 
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5 fvH, but a zero otherwise; 

6 (b) setting the value of the coefficient at coordinates (x, y) to the obtained value; and 

7 (c) repeating steps (a) and (b) until a value is set for every coefficient in the block. 
1 

1 8. The method according to claim 7 further comprising the step of setting the 

2 block coefficient at (0,0) to a zero value. 

1 9. The method according to claim 7 wherein each of the random values obtained 

2 for so long as x and y lie in a band pass region defined by the cut firequencies fnu fyL, fm and 

3 fvH follows a Gaussian random distribution. 

1 10. The method according to claim 7 wherein each of the random values is 

2 obtained by accessing an entry fi-om a pre-computed look-up table of random values. 

1 1 L The method according to claim 7 further including the step of establishing an 

2 inverse transform on the block of transformed coefficients. 

1 12. The method according to claim 1 1 further including the step of scaling the 

2 inverse transform of the block of transformed coefficients to yield a pattern of film grain 

3 samples. 

1 13. The method according to claim 1 2 further comprising the step of initializing a 

2 database of film grain patterns using a single block of transformed coefficients to create all 

3 possible film grain patterns. 

1 14. The method according to claim 1 2 further comprising the step of initializing a 

2 database of film grain patterns using multiple blocks of transformed coefficients to create all 

3 possible film grain patterns. 
1 

1 15. Apparatus for simulating film grain, comprising one of a progranuned 

2 processor and logic circuit for (1) creating a block of transformed coefficients responsive to a 

3 band pass region associated with a desired film grain pattern; (2) establishing an inverse 

4 transform for the created block of transformed coefficients; and (3) scaling the block of 
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5 inversely transformed coefficients to create a simulated pattern of film grain. 

1 16. The apparatus according to claim 1 5 wherein the one of the programmed 

2 processor or logic circuit creates the block of block transformed coefficients by (a) obtaining a 

3 value for each coefficient at coordinates (x, y) which is a random value for so long as x and y 

4 lie in a band pass defined by the cut firequencies fnu fvu and fvH, but a zero otherwise; (b) 

5 setting the value of the coefficient at coordinates (x, y) to the obtained value, and (c) repeating 

6 (a) and (c) until a value is set for every coefficient in the block. 

1 17. The apparatus according to claim 1 5 wheriein the one of the progranuned 

2 processor or logic circuit creates the block of block transfohned coefficients by setting the 

3 block coefficient at (0,0) to a zero value. 
1 

1 18. The apparatus according to claim 1 5 wherein the one of the programmed 

2 processor or logic circuit creates the block of block transformed coefficients such that each of 

3 the random values obtained for so long as x >= Thl or y >= fvL and x <= fHH and y <= fvH 

4 follows a Gaussian random distribution. 
1 

1 19. The apparatus according to claim 1 5 wherein the one of the programmed 

2 processor or logic circuit obtains each of the random values by accessing an entry firom a pre- 

3 computed look-up table of random values. 
1 

20. Apparatus for simulating film grain, comprising: 

means, for creating a block of transformed coefficients responsive to a band pass 
region associated with a desired film grain pattern; 

means for establishing an inverse transform for the created block of transformed 

5 coefficients; and 

means for scaling the block of transformed coefficients to create a simulated pattern of 
film grain. 
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